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PLEASE AMEND THE SPECIFICATION AS FOLLOWS: 

(Please replace page 10, line 25 through page 51, line 9 with the following amended 

specification.) 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT: 

While this invention is susceptible of embodiment in many different forms, there is 
shown in the drawings, and will be described herein in detail, specific embodiments thereof with 
the understanding that the present disclosure is to be considered as an exemplification of the 
principles of the invention and is not intended to limit the invention to the specific embodiments 
illustrated. 

In accordance with the teachings of the present invention, a system and methodology are 
provided for music presentation and communication. Musical compositions can be input to the 
present invention from any one or more of multiple sources, such as from prestored score 
images, live microphone, direct input from musical instruments or vocal direct performances, 
scanning in of existing printed score images (optically character recognized), cameras, visuals, 
etc. These inputs by the system are used in the selective storage, composition, communication, 
and presentation of the musical system of the present invention. The system can generate 
additional material automatically, or permit a user to modify, communicate, display and/or 
reproduce the musical compositions. 

Modification can be performed on rhythm, primary key, individual notes, chords, etc. 
The vast store of musical information stored in digital notation format and/or any video format, 
can be broadcast (analog or digital) to a local music workstation or a master controller, which 
can also be a local workstation. The master controller can be a stand alone unit, or act as a server 
as well as its own stand alone unit, or simply as a server to a plurality of other stand alone units. 
However, in the minimal configuration, only a single musical user station is needed. 

In one preferred embodiment, the workstation is provided as a music stand where the 
display presentation is a liquid crystal display (LCD). The LCD that can provide monochrome, 
gray scale or high quality color displays, depending on design and cost constraints and desires. 
Other display types can also be used. A touch-screen input can provide for simplified and 
adaptive user input selections. An optional built-in metronome function can also be provided for 
display presentation audio and/or video. A subsystem can also be optionally provided to permit 
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the music to be audibly reproduced at the workstation through a speaker or headphone jack or 
other output. 

It is well known in the art to convert user analog audio input into a digital format, ranging 
from straight Analog to Digital (e.g., A/D) conversion to processed data conversion to encoded 
digital music data, such as MIDI. Examples of MIDI include guitar input or other stringed 
instrument input through microphones or directly to MIDI-converters, or voice/non-pickup 
instruments through microphone converted to MIDI-input, or keyboard MIDI-input. Such input 
systems are commercially available from numerous companies for numerous types of interfaces 
at numerous interface levels. Similarly, numerous A/D converter subsystems are commercially 
available at chip and board solution levels (such as from Analog Devices Corporation and from 
Mattrox Systems). 

In accordance with one aspect of the present invention, the multi-dimensional music 
transformation system of the present invention also enables a user to select one or more musical 
compositions from a larger database from a plurality of musical compositions. The database can 
be stored locally within the workstation, on site, or remotely stored and transmitted to the user 
(such as over cable, wire, telephone lines, wireless (such as radio frequencies)). The user can 
also optionally edit the selected score of the composition (such as changing the key and/or any 
note and/or timing, etc.) to suit his or her taste. The score (altered (the derivative composition) 
or not (the original composition)) can then be transmitted to one or more displays such as liquid 
crystal or CRTs in the music stands of the band or orchestra. The present invention, therefore, 
provides an efficient, paperless solution to communicating, presenting (displaying), and 
optionally one or more of transposing, editing, inputting, comparative testing-teaching, 
conducting, and disseminating music to one display or a large number of displays. Each display 
can have the same, or a different, unique, customized presentation of music notation as 
appropriate per selection, responsive to a set-up by a system, automatically per predefined 
parameters, and/or to user input. The score can also be printed out if a hard copy is desired. 

As illustrated in FIG. 1, the music is stored, such as on a large hard drive or CD ROM 
jukebox, in a digital format as a music library (120). The music library (120) is coupled to a 
processor subsystem (115). Coupling can be wireless or cabled such as through a shielded cable, 
fiber optic conductor, switched connection (such as via phone lines), local, or remote. The 
processor (115) has the local storage capacity (e.g., semi- conductor memory, disk storage, etc.) 
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to hold the digitized version of the music composition transmitted to it on request from the 
library (120). The music library can be local or proximately remote from the rest of the system. 

In a wireless embodiment, the music library (120) is coupled to a communications 
subsystem (such as a radio frequency transmitter) (125) that transmits the contents of requested 
compositions from the remote music library (120) to the processor (115) through the antenna 
(104) of the transmitter. The antenna (102) at the receiver picks up the transmitted signal and the 
receiver conveys it to the processor (115). This embodiment enables the music library (120) to 
be remote and located at a great distance from the requesting site. The communications 
subsystem (125) can be a transceiver for bidirectional wireless communication, or a transmitter 
for one-way wireless communication (such as where the requests are otherwise communicated to 
the music library subsystem (120), such as via a wired connection). 

As illustrated in FIG. 1 A, a system controller, in the form of a music stand (105C) with a 
liquid crystal display, is used by an operator (e.g., performer, conductor, etc.) to select one or 
more musical compositions. FIG. 1 A illustrates two types of music workstations stands. The 
workstation stand (105C) provides certain optional features for a more full-featured stand, 
including as illustrated, speakers (140) both wireless and wired communications capability, and 
as illustrated, shows the processor with memory (115) as an external separate component. The 
music stand (105P) shows the integration of the processor and memory into the music stand 
itself, and also shows both wireless (antenna (101)) and wired connection (port (107)) to permit 
network communication. Alternatively, the conductor stand (105C) could have all or part of the 
features integrated into the music stand (105C). Depending on the function for which the music 
workstation stand will be used, some or all of the features can be provided for that stand to 
minimize costs or optimize versatility. For example, in one situation, only the teacher or 
conductor needs the full-featured, full-powered music workstation. In that case, the performers 
or students do not have a full-feature workstation, but rather a scaled-down version of the 
workstation stand. In the preferred embodiment, a user input device (110) (such as a touch 
screen, microphone, keyboard, switches, voice recognition system, visual recognition system, 
etc.) is coupled to the processor in a wired (such as over a cable or fiber optic link) or wireless 
(such as over an RF link or infrared link) manner for workstation stand (105C), or directly to the 
processor, where it is built into the system controller as workstation (105P). The user can select 
an original musical composition from the touch screen of the liquid crystal display (135). The 
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processor responds by storing that composition in the memory (115) of the local workstation of 
the user as requested. 

Using the touch sensitive LCD (135), the user can now create a derivative musical 
composition. The touch sensitive LCD allows the user to enter the musical key in which the 
original composition will be played, edit any notes desired, and select the instruments and parts 
that will be playing the composition. The composition as originally composed, and the 
derivative or modified composition can be played back to the user over speakers (140) so that he 
or she may listen (e.g., such as to observe how the changes will sound) while optionally 
permitting simultaneous viewing of the score on the presentation visual display. Once the score 
has been designated (e.g., selected, edited, etc.) to the users (e.g., conductor's) taste, the 
appropriate portions (e.g., by musical instrument) of the scores can then be transmitted for 
(optional storage and) display to the respective associated individual music workstation stands of 
the band members. 

In a preferred embodiment, each stand has an input device (110) that permits the user of 
the stand to select which instrument will be using the stand. (As discussed above, this input 
device can take the form of a touch sensitive screen or a number of buttons or switches or voice 
or audio recognition, etc.) 

In the preferred embodiment, each individual music workstation stand (105) can be 
directly and/or remotely programmed to addressably receive (and optionally to locally convert) 
and display the music score that js intended for the respective instrument type (user type) that 
will be using (is associated with) the stand. As an example, the user of the stand (or a conductor) 
can input their selection of saxophone into the user input device (110) of the workstation stand 
(105C), to program that workstation stand (105C) only to receive the musical score for the 
saxophone (see FIG. 3). Then, the musical scores for all selected parts can be independently 
broadcast to all connected workstation stands, with each individual workstation stand 
individually distinguishing and accepting only its part. Alternatively, each workstation stand can 
be individually addressed for separate broadcast reception of its own respective selected part. 
Additionally, the user of the stand can program the user input to select a musical part of a 
selected musical composition (e.g., saxophone first chair) and receive only the musical score 
intended for that chair. This same procedure can be followed for other instruments within the 
band or orchestra. Alternatively, a single music composition can be broadcast to all 
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workstations, where each workstation has local intelligence (processing and storage) to permit 
local conversion for display at each workstation for the selected instrument for each workstation. 

For wireless communications, the individual music workstation stands (105) are 
comprised of receivers (or transceivers where bidirectional communication is desired) and 
antennas (101, 103) for receiving (or transceiving) the radio frequency information from (and to) 
the master workstation (such as for the conductor). The music stand also has a display (such as 
an LCD (135)) for displaying the musical score intended for that stand. 

Referring to FIG. IB, the music workstation stands can either be identical or broken 
down into conductor stands and performer stands. A conductor stand (105CON) may have more 
functions and control than a performer stand (105PER). A performer stand (105PER) might 
only have the ability to receive and display musical scores, whereas the conductor stand 
(105CON) has the ability to select the musical score, change the key of the musical composition, 
and perform other tasks only a conductor would be permitted or required to do. 

In one embodiment, an RF antenna for the stand (105) can be built into the stand itself. 
Alternatively, instead of using RF, the performer's stand can be linked to the main (e.g., 
conductor's) stand using infrared, fiber optic cable, shielded cable, or other data transmission 
technologies. As discussed above, the communications link can be bidirectional, such as to 
facilitate requests and responses to facilitate the feedback of performance parameters or such that 
any workstation can be a master or slave, or used in combinations. 

FIG. 2A illustrates the overall operation of the music composition communication 
workstation. It begins by starting up the system (200). The system then provides a menu (201) 
that allows the user to select (205) a listing of available music compositions. The user then 
selects one or more compositions (210). If the user selects one from the menu that is locally 
stored, it directly retrieves the information. Alternatively, if it's not something locally stored, the 
system couples (e.g. will dial up or go through a database or network) to a remote storage site 
and requests and receives the selected compositions. 

Any changes that ar e d e sir e d to tho composition can bo selected at the n e xt logic block 
(215). If there are desired changes to the composition (such as to the key (255) , or note editing 
(270) , or selection of form of display or instruments (280)) , then those can be respectively 
accomplished as illustrated at blocks (255) to (2 8 5) (260). (265), (275) and (285) . 

If no changes are desired, the musical score for the composition that is selected is 
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broadcast, transmitted, or otherwise transferred to the workstation music stand (220). It is 
internally stored in the local workstation music stand. Next, the score is displayed (225) on the 
workstation display (e.g., LCD or CRT) or a video projection system. The display can also be 
part of an integral stand-alone workstation or an interconnected group of components including a 
personal computer (such as Macintosh, or DOS or Windows PC). 

The display mode selection is then made (230). This permits selection of an operational 
display mode, not simply choosing the resolution or color. The two main choices in the 
preferred embodiment are a manual mode (250) and an automated mode (240). In the automated 
mode selection (240), there are many sub-modes or options, such as the operational mode that 
permits the performer or user to do their performing without having to tend to the selection of the 
portion of the music to be displayed or the turning of pages. In the auto performance mode-as 
shown on LCD (135P) , there is provided the simultaneous displaying of the selected musical 
composition, and a display representative of the audio performance of the user, and a divergence 
signal or divergence data representative of analyzing the performance, preferably in 
approximately real-time. 

FIG. 2B illustrates the manual mode (250), which provides for user manual selection of 
functions (252). There are many functions that the user can select, even in the manual mode, 
such as hitting a button or a touch screen to cause the turning of the page of the display. Another 
function is to go back a page or to scroll forwards or backwards. For those who are vision 
impaired, another function can increase the font size of the music presentation. 

Thus, there are many manually selected functions that can be provided. While the 
manual mode can have automated functions selected, it is distinguished from the automated 
mode where control is partially predefined without user assistance. In the manual mode (250), 
the user selects any and all features that are going to be provided (some of which can be 
automated). The selected function is then processed (256). 

Next, any ongoing needs are processed (257). These needs can include any overlapping 
automated function (not otherwise inconsistent with any other selected function). 

Referring to FIG. 2C, the operation of the automated mode "A Mode" (240) is illustrated. 
First, the user selection of the desired automatic mode is detected and responded to, illustrated as 
the auto-advance mode- option (242), the training mede- option (244), the performance mode 
option (246), or any one of a number of other modes- options (248) as is described in further 
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detail hereinafter. For example, auto repeat mode can be selected by designating the start and 
stop points, and the number of times to repeat a "looped" portion (or portions) of the displayed 
musical composition. Marching band mode (auto-advance based on metronome function, 
conductor control, etc), auto-compose mode, and many others can also be implemented. The 
order of selection of auto-advance, training, or performance mode is arbitrary, and the user can 
alternatively decide from a menu where all are simultaneously presented as choices. 

The display can advance the music by page option, or by a user selection of one of many 
options (e.g., scrolling, tablature, video graphic tutorial display, etc.). 

Referring to FIG. 2D, the automated mode 1 for auto-advance operation (242) of FIG. 2C 
is illustrated, where the user has selected an auto-advance performance mode. In this mode "A 
Mode 1" (271243), the system tracks the performance by the user of the composition to the score 
(272). Performance refers to the actual performance by an individual person (or people) who is 
(are) reading the musical score upon which the performance is based. Whether that score is in 
tablature format, staff and clef and note notation, or some other format, the system generates 
appropriate signals to permit comparison of the user's performance to the musical score. 

Based on a comparison, a decision is made pursuant to selection criteria programmed into 
the system (such as the rate at which the piece is being played, the time signature, the tempo, the 
rhythm, and the advancement of the music on the available display), the display presentation is 
advanced ( 27 4 and 278). In some cases, the music might move backwards, such as with D.S. 
Coda. The presentation of the display tracks the performance to permit smooth, uninterrupted 
playing or singing. The capability can be provided for the user to override this auto-advance, 
such as for practicing where it is desired to keep going back over sections. In this case, a user 
override option (276) is permitted to alter the automated operation. Upon cessation of user 
override, the system can be programmed to stop, to automatically return to the regular auto- 
advance mode, or to process other auto-modes (270279) of FIG. 2C. 

Referring to FIG. 2E, the automated mode "A Mode 2" (244245) operation of FIG. 2C is 
illustrated corresponding to the training mode. In this mode, the system tracks the performance 
(280) of the individual user to the composition score, primarily for the purpose of permitting a 
critical analysis and comparison of the performance to the score (282). This analysis determines 
divergence from the selected musical score, and reveals errors or deviations from desired 
performance goals (e.g. match of timing of notes, duration of notes, pitch of notes, etc.), and to 
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display those errors (284) (such as by audio or video means). Predefined performance goals 
provide the knowledge basis for expert system based analysis. 

The system can then generate a graded score (286) indicating errors, and can present it in 
numerous formats such as histograms, frequency of errors, spacing of errors, etc. Identification 
of when the errors occur (e.g., only when going from slow to fast, or fast to slow), absolute 
position within the score and so forth, are also tracked and reported. Other expert system rules 
can be provided by music teachers which give the necessary parameters for modeling expert 
system reasoning, as well as guidance and suggestions on how to correct problems such as via 
display text, graphics, audio, etc. 

The comparison of the performance to the score in the training mode is for the purpose of 
detecting the performer's compliance to parameters (such as the tempo, rhythm, filter, parameter, 
pitch, tonality, and other features that are adaptable or can be modified by performers). This 
parameter information is available and published in numerous forms. Thus, having provided this 
core set of parameters, the system can thereafter perform the training automated mode. 

As illustrated in FIG. 2F, automated mode 3 "A Mode 3" is the performance mode 
(246 247} of FIG. 2C t). In this mode, the operation is as in automated mode 1 (auto-advance 
mode) except that no user override is permitted. Its primary purpose is to accompany the 
performer during the entire performance of a score as an automated page turner. The tracking of 
the "page turning" to the performance can optionally be based on inputs or criteria independent 
of a performer's actual performance input (e.g., microphone), such as a built-in metronome 
clock, a central control (e.g., a conductor or special user input), etc. Additionally, performance 
characteristics can be tracked, computed, and reported as in the teaching and training mode. 
Training feedback can optionally be provided real-time, or subsequent to completion of 
performance, to assist the performer as in the training mode. Alternatively, the score can be 
presented in a moving score mode (e.g., vertically, horizontally, or otherwise) or linear 
presentation as opposed to a real page turning display. 

FIG. 2G shows the operation of automated mode 4 ("A Mode 4"} (249) of FIG. 2C- which 
provides for the processing of other automated functions selected by the system. These modes 
can include conductor mode, karaoki mode, etc.T 

In conductor mode, a conductor can control communications of signals to his or her 
performer (such as "increase volume", or "increase tempo", or "play staccato"). Icons can be 
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provided where the conductor simply touches a touch screen (or other input mechanisms) to 
supplement his hand and body motions to permit more effective communication with the 
performers. Alternatively, as illustrated in FIGS. 9 and 10, in a more advanced system version, 
the conductor's movements are first learned by a monitoring system, based on user definition 
and assignment of meanings for movement to provide an expert knowledge database. 

This system provides for tracking of movement input such as in FIG. 10 via video camera 
(1005) input of the conductor (1015) against a backdrop (e.g., blue screen) (1010) is processed 
by video processing unit (1020), or, as shown in FIG. 9, via body glove technology (gloves (935) 
or sensors (944) or sensor clothing (940) or head or eye movement tracking sensor (930) (such as 
used in virtual reality, flight simulation, avionics equipments (such as jets and space travel), and 
sports players for analyzing movement) to provide the necessary movement input. This 
movement input is analyzed utilizing the expert knowledge database to automatically generate a 
display (video and/or audio) to provide local visual and/or audio reinforcement on the local 
display (such as overlaying on a then unused portion of the music score display as a picture in a 
picture) to permit audio and video reinforcement of the conductor's body language. Thus, "a 
hush" body language signal that is directed towards a particular section of the orchestra would 
automatically be interpreted to cause the system to indicate, and only on that particular section's 
respective displays, a message (e.g., big face with a finger in front of it making a hush sound 
with a "hush" sound simultaneously output from a speaker). The conductor mode provides many 
benefits to performance and communication. 

For all these automated modes (e.g., A Modes 1, 2, 3, 4), training feedback can be 
provided real time or subsequent to performance at either or both of the performer's workstation 
and a second (e.g., teacher's) workstation. 

The advantages of electronic music composition, communication and display are many. 
In addition to those discussed elsewhere herein, a capability exists for expert system based 
artificial intelligent type assistance where the expert system assists in many of the functions 
performed in musical composition and performance. For example, in the Auto-Compose Mode, 
if the words need to be changed to match the meter, equivalent terms can be chosen from the 
many sources such as thesaurus, dictionaries, rhyming dictionaries, encyclopedias, etc., to assist 
as well. Phrases from poetry, selected and indexed by content or topic can be re-expressed to 
create new works. Drum and rhythm section accompaniment can be expertly suggested, as well 
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as harmonies, melody lines to accompany chords, chord progressions to accompany melodies, 
harmonies to accompany a melody, and suggested musical instrument groupings to support a 
particular sound, rhythm, style, tonal quality, etc. 

The expert system can be built from commercially available technology, including 
component hardware systems with supporting software, as well as commercially available 
software packages which operate on commodity-type personal and business computers such as 
the Macintosh by Apple Computer, Windows and DOS machines based on the X86 and Pentium 
processor technology of Intel, technology based on the Power PC and 68XXX processor by 
Motorola, DEC PDP-1 1 technology, Sun workstations, etc. Custom microcomputer or DSP 
based system architecture on a chip can also be constructed, as well as ASICs, custom or semi- 
custom logic. 

The system can be designed to take advantage of expert system design knowledge. A 
database of rules and facts are provided, and accumulated over time by the system in a self-learn 
mode. The expert system itself has the necessary logic to probe the user, monitor the 
performance, and apply the rules to provide feedback and reports to the user of skill level, errors, 
automated performance display, etc., starting with a base defined set of rules, instructions, and a 
knowledge database specific to music. 

The form of the musical score communication can be easily shaped to fit needs. One 
example is MIDI (Musical Instrument Digital Interface standard) which has advantages such as 
of bandwidth of storage used, is widely available commercially, is standardized, etc. However, 
signal processing, text, icon-based, object based, and various other forms of storage, user 
interface, and processing can also be applied to more specific applications of product. 

FIG. 3 illustrates one embodiment of an LCD display used for input control and for 
displaying the information from the processor and memory. In the preferred embodiment, this 
LCD is a touch sensitive screen enabling the functions associated with each displayed button to 
change, and also for the displayed buttons to be moved around the screen, depending on the 
function to be activated. The musical score may be edited by the conductor, such as by touching 
the individual note after which he is presented with a number of notes to replace the touched 
note. The lower portion of the screen displays instruments from which the conductor can select 
which instrument will be playing the composition. After a button on this screen has been 
touched, a number of sub-screens may come up, each with their own individual touch sensitive 
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areas and functions to be activated by those areas. Alternatively, in addition to or instead of the 
touch screen, the system can provide input via separate key switches, voice recognition, etc. 

As an example, if the conductor touches the transmit key on the main screen, he will be 
presented with a screen showing all of the instruments that he has selected for that piece and a 
button labeled "ALL". He may now transmit to each individual music stand or by depressing the 
"ALL" area, transmit to the entire orchestra. 

The music library can be contained ("stored") on non- volatile storage either locally or at 
a remote central site containing the entire (or a subset) database of all possible music (that is then 
downloaded to local storage on request, either real-time at performance time or in advance.) 

Alternatively, the music library can be provided on storage medium that can be easily 
transported and used on site locally with the presentation system. Thus, for example, disk drives, 
cartridges, FLASH RAM cards, plug-in memory modules, or a CD-ROM or multiple CD-ROMs 
in a CD-ROM changer can be used to store and contain massive data libraries on musical 
compositions. While this would be a more expensive route than shared use of a central library, 
requiring each musical group to obtain libraries on all possible compositions they may want, it 
has the advantage of speed, flexibility, no need for communication with a separate remote 
source, and creates a whole new mass marketing area (such as for CDs or Digital Audio Tape 
(DATs)). Another way of utilizing this technology is to maintain a history of music used, either 
with the remote music library or local music library. This could be done for many reasons, 
including copyright royalty assessment, determining a history of musical performances and 
requests for future use in determining performance itineraries, etc. Alternatively, a hybrid of 
locally stored and centrally shared libraries can be utilized to optimize cost, speed and flexibility 
benefits. 

In accordance with another aspect of the present invention, each display workstation can 
also provide the ability to convert performed musical compositions into annotated musical 
compositions, generating the appropriate musical notation (e.g., staff, tablature, MIDI), notes, 
time signature, key, instrument, or user type, etc. 

The display workstation can be implemented as a totally self-contained workstation, 
where each workstation contains its own processing subsystem, optional communications 
interface (such as wireless or cable) for network use, input/output interface including one or 
more of a user input keypad, a speaker, a microphone, joysticks, push buttons, etc. Each of the 

-13- 



9/2/2005, 1757RCE2Appv2 



PATENT APPLICATION 
Serial Number: 09/492,218 
Attorney Docket Number: STD 1757 RCE 

stand alone workstations can then operate with a local database or couple to a shared music 
database as illustrated in FIG. 4. 

The stand alone workstation(s) (105), are coupled to the shared database interface (405), 
and can either couple remotely (e.g., via phone lines) to the remote shared music database or to a 
local shared (410) or dedicated music database-(4i0). The shared music database (410) can 
either be primarily a storage means (e.g., hard disk or CD-ROM), or can include a processing 
subsystem (420) for local intelligence. In one embodiment, the stand alone music workstation 
includes the shared music database (410) and interface (405), non- volatile local storage medium 
for the shared databases (410), and a local processing subsystem (420), and can operate 
completely stand-alone. In an alternate embodiment of this stand-alone device, the shared 
database interface is contained in the stand-alone workstation (but not the shared music database 
or processing subsystem), and provides capability for communication with a stored database 
(410) remote from the stand-alone device. 

In either of these embodiments, an alternate additional embodiment provides capability 
for each stand-alone workstation to function as a master stand-alone, or a master or slave 
workstation within a workstation set including multiple stand-alone workstations, wherein one is 
designated master and the rest are designated slaves. The slave workstations in this 
configuration receive communication of music compositions to be displayed from the master 
workstation, thereby permitting one shared music database to be communicated among all 
workstations which are a part of the group. It is to be appreciated that the shared music database 
function can be distributed in many different ways among the workstations, or separable from 
and independent from the workstations. The choice is simply one of design, and the illustration 
herein should not be taken in a limiting manner. 

In one embodiment, the master workstation has complete control over the slave 
workstation. Anything displayed on the master workstation is also displayed on the slave 
workstation. It is also possible for the user to mask certain portions of the display of the master 
workstation before it is displayed on the slave workstation. In this manner, the conductor, using 
the master workstation, can transmit to the slave workstations only that information that is 
required by the orchestra members. 

In an alternate embodiment, the slave workstation communicates performance parameters 
or deviation signals to the master workstation, for error analysis feedback. 
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In accordance with another aspect of the present invention, means are provided wherein a 
plurality of individual workstations are coupled together in the network configuration to provide 
for networked communication of musical performance data wherein each of the individual music 
workstations provides for capturing the performance data for a users performance and 
communicating that performance data to a master or a conductor workstation which synchronizes 
and combines the plurality of individual workstations performance data to create a composite 
virtual performance data output which is recommunicated back to all of the individual 
workstations in approximately real time, so that the individual workstations can receive the 
composite virtual performance data and provide an audio output (and/or visual presentation) of 
the combined composite virtual performance data including all of the individual workstations 
users performances. While the networking can be used in conjunction with other features and 
embodiments of the present invention, including communication of musical compositions for 
display, and other musical performance data analysis and data communication, as well as inter- 
musician interpersonal communication, the networked embodiment of the present invention 
permits synchronized virtual performance thereby permitting multiple remotely located 
individual workstations to physically separately perform with the benefit of hearing in 
approximately real time the combined result of all the performances at the individual 
workstations with their own individual performance. 

In accordance with another aspect of the present invention, means are provided to permit 
a user of the music workstation to accomplish a transposition of a musical composition in pitch, 
tempo, and otherwise. In a preferred embodiment, the lead voice or instrument can audibly 
indicate the key via the microphone input or via another type of input stimulus. The workstation 
can analyze the user input, determine the key, pitch and tempo for a musical composition being 
partially performed by the user, and adjust and transform the composition to be displayed in the 
new user desired key, pitch, tempo, etc., either solely for use on that workstation, or 
communication for use on one or more other workstations. In a networked version, this user 
input can also be communicated to other workstations for use by one or more of the workstations 
in transposing, or communicated to a master workstation, which transposes and rebroadcasts the 
transposed composition. 

Alternatively, the user can input the pitch, tempo, and key via the user input (e.g. keypad, 
joystick, push buttons, voice recognition, playing of an instrument, etc.) and the system performs 
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the transformation and displays (and/or prints out and/or audibly performs) the modified 
transformed composition for the user. Additionally, where a musical composition is written for 
one instrument and a different or additional instrument version is desired for simultaneous 
performance, the user can indicate the other instruments via the user input, and the system will 
generate the appropriate displays. The workstation can also provide an audio output of the 
transformed musical composition, either for the individual additional instrument or voice 
transform and present it, or for the composite of additional versions and the original version, to 
hear the blended piece. 

Referring to FIG. 5, a music communication system is illustrated comprising multiple 
workstations (500) each comprising a display (510), user input such as a keypad (522), a joystick 
(524), push buttons (525 & 526), a microphone (527), and a speaker (528). The workstation also 
includes communication interface means such as a wireless interface (532> including an antenna 
(531), or alternatively or additionally a wired or cabled communication interface (540). Each 
workstation further includes a local microcomputer subsystem (550) that provides local 
intelligence and management of functions in the workstation. 

In the networked embodiment, where multiple physically separate locations each having 
one or more individual workstations provide for communication of performance data from the 
individual workstations and presentation by the individual workstations of the combined virtual 
performance data at the individual workstation, to provide for a virtual performance. In this 
case, the communications interface would utilize slightly different structure, such as a phone 
modem (analog modem), a cable modem, ISDN as between locations, etc. 

Communications interfaces of various types are well known and commercially available. 
At the present time, they are available for purchase at the chip, board, or system level. In fact, 
many single chip microcomputers include communications interface capabilities, wired or 
wireless. 

The workstation further includes an optional musical instrument input (562) and a 
musical instrument output (564) that permit the coupling of a musical instrument via a musical 
instrument interface (570) directly to the workstation. Thus, a keyboard, electric guitar through 
appropriate input, or a microphone input through the interface (570) permits instruments or 
voices to be directly input to the workstation for direct input independent of the microphone 
(527). 
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The instrument output permits coupling of the instrument input signal, either directly fed 
through or as modified by the workstation for output to the appropriate public address or 
amplification and presentation system or separate analysis system. The workstations are coupled 
either via wired or wireless communication to a processor subsystem (580) that includes a 
processor, non-volatile memory, read/write memory and an interface to a non-volatile storage 
medium (582). 

The processor subsystem (580) includes an appropriate communications interface, such 
as a communications interface (540) for wired interface or (532) for wireless interface including 
antenna (533). The processor subsystem couples to a non-volatile storage medium (582) 
containing, among other things, application programs, transformation programs, and either a 
shared music library interface application program or the actual shared music library and access 
program. 

As described above, the processor subsystem (580) and non-volatile storage (582) music 
library can be built directly into one of the music workstations (500) to be a master, with the 
other workstations being slaves, that can either include the processor subsystem and non-volatile 
storage or can be lower cost dummy slave terminals. As illustrated in FIG. 6, a first master 
workstation (£00600) provides a basic workstation subsystem (200 610) plus contains the 
processor subsystem (2*0680) and non- volatile storage system (285 685) as a part thereof so as to 
provide a complete stand alone music communication system, and be capable of acting as a 
master or master/slave. This master workstation(s) (300 600) can function as a stand alone, or 
can couple to one or more other workstations, including one or more masters (300 600) and/or 
one or more non-master workstations (105). 

The multiple connected workstations can operate as stand alone workstations using their 
local intelligence for displaying downloaded or resident music compositions. They can also 
interact in a master/slave linked environment, where one of the master workstations (300600) 
asserts a master status, and all other interconnected workstations, whether workstations (105) or 
master/slave workstations (300 600) operate in a slave mode coupled to independent on the 
designated master. Additionally, masters can communicate between each other for a 
master/master network configuration. 

Alternatively, the multiple connected workstations can operate together in a networked 
virtual performance mode, or in a networked communications mode. A dedicated stand alone or 
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distributed master workstation architecture can provide for the coordination and combination and 
synchronization of the multiple individual performance data outputs into a combined virtual 
performance data output. 

Referring to FIG. 7, an alternate embodiment of the present invention is provided where 
one or more workstations (105) include, at a minimum, a display of the music notation. These 
workstations are coupled to a master music communications controller (445 715) that provides 
for a separate user input (434 711) which provides input interface, such as to a MIDI status 
stream, computer data links (such as RS232, modem data link) etc. that designate requested 
musical compositions, transformations, and display requests for various ones of the coupled 
workstations. 

In an alternative embodiment, the master music communications controller 444- (715) 
provides for additional functionality including virtual performance mode, wherein the input 
interface (such as the MIDI stream, computer data links, etc.) provide one or more of musical 
compositions data for display, transformation information, display requests, user individual 
performance data, and wherein the workstations respond to the master music communications 
controller to couple their individual performance data and receive back the combined virtual 
performance data. 

The workstations (105) access the music database storage means (420 720) that provides 
the data for the requested music composition via the master controller (415715). The master 
controller (415715) displays both the requested music composition as well as user interface 
communication for the music communication system to be displayed on either a dedicated 
display (416 716) or on one of the workstations (105) as designated by the master controller 
(415715). The music database (420 720) can either be local, or can be via a data link (e.g., phone 
line, RF, otherwise). In one embodiment, a motion sensor subsystem (422 722) monitors motion 
of a target person and responds in accordance with predefined movement interpretation 
characteristics parameters, such as for a conductor. 

In a preferred embodiment, the user input means (411 711) is comprised of a key switch 
device, such as a touch membrane keypad or capacitance touch surface. Alternatively, in one 
preferred embodiment, the user input is provided via a touch screen technology. Touch screen 
technology permits the display of user interactive icons and legends including text and graphics 
making possible unlimited customization of user input structure according to task needs. Thus, 
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specific switches or sequences of touches to the touch screen can be associated with common use 
icons from the task being performed in conjunction with words to provide ultimate clarity. User 
error is virtually eliminated, with the aid of automatic entry error detection, such as defined 
fields, mandatory fields, etc. 

Alternatively, the microphone input (522727) can provide for coupling of user speech to 
a processor subsystem (580780) that uses any of a number of commercially available and well 
known speech recognition algorithms. These algorithms provide for speech recognition input 
control, either solely or as a supplement to touch screen or other tactile input mechanisms. 

In a deluxe embodiment, an output (423 -721) is provided that permits coupling of an 
external display, such as a color monitor, projection unit, or other display presentation system 
including one or more of audio, visual, and audiovisual. 

Additionally, the display presentation output of the workstation can provide for an audio 
output presentation of the musical performance, either generated by the work station responsive 
to the music composition data, to the users performance, to the combined virtual performance 
data, or responsive to an external source. Additionally, a visual or audio visual presentation can 
be provided to provide information feedback to the user on both their individual performance as 
well as from and between individual workstations and/or the master controller or conductor 
workstation. 

In accordance with another aspect of the present invention, means are provided for 
moving through the printed (displayed) notation of the music in synchronization with the live 
performance from the displayed musical notation. 

Musical notation is used, in the generic sense, to refer to any way of conveying musical 
performance instructions including but not limited to common musical notation with staffs, 
notes, sharps, flats, and clefs, extending to written instructions in text form to supplement this or 
supplant or partially replace, as well as alternate forms of expression such as chord charts, words 
and chords (letters), tablature, any video, graphic, audio, audiovisual or other display 
presentation or combination of the aforementioned types of presentations. 

An annoyance in performing music using any written notation (whether on paper or 
displayed on a presentation apparatus such as a screen) is smoothly performing music and timing 
"flips of pages" (or the communication of change to a third party, such as touching a key or 
button). This is especially true when the user is marching and both hands are required 
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simultaneously. 

In accordance with one aspect of the present invention, means are provided to accept 
inputs from one or more sources that initiates a "page turn". Types of inputs include 
conventional touch input apparatus (such as key switches or capacitive touch pads), motion 
sensing gear, and automatically when operating in the operational mode of Auto Mode. The 
motion sensing gear can be for a portion of the performer's body, such as a head tilt sensor or an 
optical eye movement sensor, etc. 

Additional types of inputs that can initiate a "page turn" include voice or sound 
recognition apparatus built into the microcontroller system. This apparatus has the ability to use 
pattern recognition specific to the sound or user voice and words being said (for extremely high 
accuracy). Of course, any type of user actuated device such as a foot or hand switch, or head 
motion device, or sound or voice recognition system, in a preferred embodiment, is selectively 
permitted to control the override of the normal progression of the music's play. 

The override may cause the progression to go backwards or forwards in the music score 
irrespective of the normal reading of it. The performance mode AutoMode blocks the user 
override to permit performance according to proper material timing and either progresses 
responsive to the music composition data timing, or in an optional embodiment, to the performer. 
This automatically moves through the musical score as written and preferably shows an 
indication of metronome time and an indication of the proper place in the score where the 
performer should be for that instrument at any specific time. This is especially valuable in a 
conductor mode of networked communication, where a conductor couples to one or more music 
workstations. 

The user's performance can be compared to the score, and feedback can be provided to 
the performer as to the quality of their performance. 

In the performance monitor mode, for a single user or multiple users, the user (or a 
remote teacher or conductor) can indicate the rate at which he feels the performer should be 
performing. A microphone input on the music workstation samples the user's actual 
performance and permits providing a graphical mapping (for the user or teacher/conductor) 
showing the relative synchronization of the performer's actual performance versus the 
conductor's desired performance. 

In an alternate automatic advanced mode, the display of the music composition is 
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synchronized to the performers actual performance. Thus, rather than simply indicating visually 
for the teacher/conductor or user what their relative performance was to the written displayed 
musical composition, the relative performer to written music synchronization information can be 
utilized to adjust the display rate of the actual musical composition to match that of the 
performer. 

With use of appropriate sound baffling, a plurality of instruments can simultaneously be 
monitored and controlled by the conductor, so long as each instrument's output sound pattern is 
communicated directly to a respective workstation. The output of each of the workstations can 
then be coupled to the conductor's master workstation for further analysis and processing. 

A workstation for an oboe may have a built in slide boom with a featherweight 
microphone to be able to receive sound input from the oboe. Electric instruments, such as 
guitars, keyboards, and other electrical analog signal sources can be fed directly to a line input 
that is appropriately buffered and filtered. Signal input can also be accommodated through a 
MIDI-interface subsystem that permits both utilization of data in workstation to workstation 
communications and utilization of MIDI-output at the station where the data was input. 

For networked virtual performance, and for one aspect of output display presentation, the 
utilization of MIDI input and MIDI output, at each of the individual workstations and at the 
master controller workstation, permits the capture of the user performance and conversion to 
individual performance data which includes time synchronization information which can be 
communicated to the master workstation, which synchronizes and combines the individual 
performance data to generate combined virtual performance data which is then communicated 
back to the individual workstations which utilizing their MIDI output interfaces provide for 
display presentation ( e.g. audio output) of the combined virtual performance data. Additionally, 
even where virtual performance mode is not selected, the provision of MIDI interface input and 
output on the workstations have other multiple beneficial users as discussed elsewhere herein. 
Additionally, other types of user performance to user performance data input devices and 
transducers can be utilized, as are well known commercially available including variations of 
analog digital converter input devices, audio signal capture, etc. 

By combining the conductor and performance mode operations, the workstation can be 
enhanced to provide training and comparison of performance to actual music. 

Some music is only available in annotated forms where there is not an existing signal 
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showing proper synchronization of the signals. Thus, a controller subsystem (such as (580780)) 
provides for real time conversion and analysis of syntax of the music notation, in conjunction 
with a precision clock metronome, and provides an indicator (such as color or other highlighting 
or holding or accentuating) of the relative timing of the performance relative to a place in the 
sheet music (or other form of musical notation). 

Existing forms of music notation can be converted manually, or can be converted 
automatically by scanning in sheet music, recognizing (using optical character recognition) the 
various elements of the music, and facets and specifics of the syntax in the form of notation 
including its constants and variables and protocols, and integrating via an artificial intelligence 
type expert system that notates, highlights, and accentuates via synchronized metronoming of 
time signature to music. Any of a variety of other means of converting music can also be used, 
such as direct input of musical performance signals processed via software that converts it into 
musical notation. Such software is commercially available, such as from ARS NOVA, Wildcat 
Canyon Software, Mark of the Unicorn, Inc., and Passport Designs, Inc. 

Since the music notation is now in computer usable form, it is now an easy task to 
communicate, display, compose, alter, and transpose music (such as in key, for types of 
instruments or voice parts, and harmonies) via well known techniques. 

Additionally, where the user input is converted into user performance data for the 
workstation, the users individual performance data is also now in computer usable form and if 
appropriately converted from performance to performance data, includes appropriate 
synchronization data relative to master controller performance synchronization signal. Thus, 
both the music notation is in computer usable form (making it easy to display, alter and 
analyze/compare), and the users performance is in computer usable form (digital individual 
performance data), it is possible to provide intelligent operation and analysis and utilization of 
both the music composition information and the user performance information, to provide for 
various automated modes and features to the users 

Implementation can also be in a custom design comprised of a microprocessor, non- 
volatile storage memory, read/write memory, and whatever additional peripheral circuitry is 
needed (such as are available in ASICs, or single chip microcomputer chip sets including CPUs, 
DSPs, A/D, and other peripheral support circuitry). These single or multiple chip solutions can 
be utilized to create a dedicated system to perform complete music workstations performance 
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criteria to support an extremely low cost, high volume music workstation solution. 

A new form of communication is created in that both the process of communicating via 
standard notation is respected and adhered to, while at the same time permitting interaction and 
communication of music media signals ranging from simple analog, digitized (analog to digital 
converted), individual performance data (representative of the user's performance), and can 
optionally include timing/synchronization data. 

A multi CD ROM changer accommodates indexed storage of hundreds of thousands to 
millions of musical compositions to permit complete stand alone operation of the user music 
workstation. Alternatively, an optional built-in or external modem can be provided to permit 
intercommunication with a remote central music database management system that permits both 
communication and down loading (and disconnect) for stand alone operation. Thus the 
workstation can stay on-line, pulling up music as needed, or can request a single or multiple 
pieces of musical works be provided to it, that are then downloaded from the central database 
manager. The user workstation then disconnects from the music database management system, 
and thereafter operates stand alone where all desired music is stored locally in storage 
(preferably non- volatile). Storage can be semiconductor, magnetic, optical or any other medium. 

The use of virtual reality technology, including motion sensors and body gloves, permits 
monitoring of various other things (as shown in FIG. 9). For example, as shown in FIG. 10, a 
camera in conjunction with analysis logic, such as expert software, can monitor motion of role 
model behavior and compare performer behavior. Hand, finger, arm, leg, eye, head, body, and 
mouth movements can all be monitored and constructive critical feedback can be accumulated, 
analyzed, and fed back to the user or teacher, for performer training, or performances, or for 
conductor communication. 

The input of monitored movement data is provided to the user workstation, permitting 
precise mechanics training such as finger position, the angle of striking of strings relative to the 
neck of a violin or guitar, or they can be used to permit the virtual performance of music by a 
performer using a virtual link apparatus such as a virtual reality glove and head movement 
detection apparatus. The user can then perform a piece with their own personalization without 
any musical instrument in fact. 

For example, the guitar portion for a piece of music could be displayed in notation form 
and actually performed according to the timing of movements of the user's fingers (either actual 
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fret positions, or only timing information). To add further reality, a mock guitar, keyboard, flute, 
or other instrument can be used and combined with virtual effects to provide for music 
performance and personalization. Thus, for entertainment purposes, users could perform as part 
of a symphony orchestra playing a violin portion. If they performed out of time, they would hear 
their instrument's performance out of synch with the rest of the orchestra's performance. 

There are numerous ways to embody the conductor movement interpretation system. As 
illustrated in FIGS. 9 and 10, one is utilizing the body movement detection apparatus prevalent 
in virtual reality, sports medicine, etc., as discussed above, to identify specific movement 
patterns or signal parameters associated with certain movement patterns, to initiate a display 
presentation, audio, video, or audiovisual to provide a presentation associated with movement of 
the conductor. Alternatively, other techniques can be used such as taking the video feed from a 
video camera or other video source (e.g. VCR) and having the conductor interpret his 
movements and assign them unique meanings, to create a lexicon of his movements and 
corresponding meaning. 

For example, rapid downward movements of the hand from up to down, in a certain 
manner, indicate "decrease the volume." When he points at a particular section at the same time 
as he is doing that, he is indicating that only that orchestra section is to reduce volume. In this 
manner, either camera input of movements, glove sensing of movements, or other techniques 
(such as audio, ultrasonic, etc.) can be used to track movement to permit associated meanings to 
be attached or indexed to particular signal parameters or parametric signals of the meaning of the 
movement parameters as provided by the conductor input device. For example, in the case of the 
virtual reality glove, that input would be the signal output of the glove as interpreted by 
associated software in a processor (such as a PC or a MAC). Alternatively, for example, in the 
case of video camera input, it could be pattern recognition or analog signal comparison to 
determine the presence of certain signal patterns indicating to the system to initiate automatic 
communication of a conductor presentation. In so doing, the conductor is able to rapidly convey 
his meaning, focus it to a particular group of instruments, and be done with it. He doesn't have 
to focus very long or concentrate to make sure they've gotten his signal. Instead he can focus on 
listening to see if they got his message. 

FIG. 8 illustrates an alternate embodiment of the present invention. In this embodiment, 
the workstations are remote units (801-803) used by a member of a marching band. Each of the 
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remote units (801-803) are equipped with receivers (810- 812) that receive musical compositions 
transmitted to them. Remote units controllers (820-822) control the operation of the remote unit 
(801-803). The musical composition is displayed on the remote unit's displays (830-832) which 
displays can be an LCD multiple line display providing low cost, low power usage, and high 
visibility/readability, and with Auto Advance Mode, the display automatically scrolls as the 
music is to be performed. 

Each remote unit (801-803) can be mounted on the instrument on or in place of the lyre. 
The remote unit's antenna (840-842) can be separate from or built into the remote unit or the 
lyre. 

A transportable main unit (850) is used to transmit musical compositions to the remote 
units (801-803). The transportable main unit (850) is comprised of a controller (806) for 
controlling the transportable main unit (850), a music database storage medium (805) containing 
the data for the musical compositions to be played by the band, and a transmitter (804) for 
transmitting the musical compositions to the remote units (801-803). This main unit can be in 
the form of a suitcase or briefcase size item. The main unit can also be provided built into a van 
that is driven around with the band or as a small self-contained portable unit. In accordance with 
this embodiment, the band can play a virtually unlimited number of musical compositions 
without the problem of carrying the music with them in paper form. It also relieves the band 
members of the problems of changing music and changing pages while marching. As discussed 
in the above embodiments, in the performance mode, the musical score is automatically scrolled 
across the screen display (830-832). Additionally, a keyboard and/or microphone can be 
attached to the transportable main unit allowing the conductor to send messages to the remote 
units via displays (830-832) or via a speaker associated with units (801-803). This allows the 
conductor to send instructions to the band (such as to take a certain route, or play at different 
volumes or speeds). With bidirectional communications and user performance feedback, the 
conductor can also monitor for errors. 

FIG. 9 illustrates a conductor, stage hand, or other person with a sensor glove on each 
hand (935) and a head and eye movement monitor (930). The figure also illustrates the 
conductor wearing full body sensor equipment (940). Either embodiment or a combined 
embodiment can be used to map body movements. If only the gloves (935) or body sensors 
(944) are used, the movement of the glove or sensors can be captured by a video system, as 
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illustrated in FIG. 10. 

Other methods that capture motion rely on specialized sensors (944) placed on a 
performer's joints, such as via a sensor body suit (940). Once motion has been filmed or 
analyzed, a data set is produced to interpret that movement into Cartesian coordinates. These 
coordinates provide the spatial location of each of those markers. This information is then 
cleaned up and input to an animation package. 

FIG. 10 illustrates a video camera (1005) and a standing conductor (1015) (or performing 
musician to be tracked or virtually linked to perform), with or without a blue screen (1010) 
behind him. The video camera (1005) feeds a video signal to the video processing system (1020) 
that utilizes signal processing to provide signal pattern recognition capability. The blue in the 
screen is filtered out in the signal processing such as by an Ultimatte process. 

In one embodiment, the conductor is wearing a sensor equipped body suit (940) and 
gloves (935) of FIG. 9. In another embodiment, the conductor is wearing only the sensor 
equipped gloves (935) of FIG. 9. In still another embodiment, the conductor's movements are 
picked up by the video camera (1005) and processed without a sensor suit. 

Simple things, like looking for the conductor's rapid hand movements, focusing on 
specific hand movement areas, facial and head movement, arm movements, and body language 
can all be programmed into the recognition knowledge base. Some of the technology for 
complete mapping of body movement that exists in making video games of today are illustrated 
in Video Systems magazine, page 42, October 1995, Vol. 21, No. 11, and NEXT Generation 
magazine, pages 49-54, October 1995, both incorporated herein by reference. 

In any event, having now obtained knowledge related to recognition of the movements, 
the system can interpret them and utilize them to convey presentation information to the 
ensemble or orchestra or studio members, or to analyze a performer's movements, or to permit a 
virtual performance. One example would be a large screen television or multiple large screen 
televisions for viewing by the members of the viewing group. Alternatively, each music stand 
could provide for a picture in picture display of special movements of the conductor in areas of 
the display where music is not currently being played. Since the stand can have the intelligence 
to compare the performed music to the played music, that embodiment permits display of the 
message in portions of the music display area which have already been performed or are not 
going to be performed for some time (e.g., at least ten seconds in either direction; other criteria 
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could alternatively be used, and can be set up for desired characteristics of the performing 
environment). 

Voice recognition and response to conductor commentary can supplement the system. 
The system could record the message, interpret to whom the conductor directed the message and 
convey it audibly or translate it into a text or icon display as a part of the system's audiovisual 
presentation. 

Referring to FIG. 1 1 A, an alternative embodiment of the automated mode "A Mode" 
1140.(240) of FIG. 2C) is illustrated. The illustrated process for the automated Mode 340- 1140 
of FIG. 1 1 A includes process steps for the Auto Advance A Mode 1 1112 , Training A Mode 2 
1145 , Performance Mode A Mode 3 1147 which are equivalent to the illustrated A Modes 1, 2, 
and 3 of FIG. 2C. However, as shown in FIG. 1 1 A, at step 1148, the decision is made whether 
the conductor automated mode is selected, and if a "yes" decision, then automated Mode 4 is 
entered at Step 1149, and if a "no" decision, then logic goes to the networked virtual 
performance mode A Mode 5, as decided at decision logic step 1150, and responsive to the 
selection of the networked virtual performance mode, A Mode 5 (1151) is executed. If 
networked mode A Mode 5 is not selected, then processing proceeds to decision logic step 1152 
to decide whether the synchronization of the display to the performance (Mode A 6) has been 
selected, and if so, then the processing proceeds at step 1153 to provide A Mode 6 functions. 
The automated mode processing ends at step 1154. It is to be noted that the automated modes 
can be cumulatively selected, so that the networked virtual performance can include any or all of 
the other automated modes, including A Modes 1, 2, 3, 4, 5, and 6, or other A Modes which may 
be provided. It should also be noted that the tracking of the performance by the user relative to 
the presentation of the musical composition (step 272 of FIG. 2D) provides the first part of the 
logic necessary for the A Mode 6 (synchronization of display to performance), as illustrated in 
FIG. 13 in further detail. 

Referring to FIG. 1 IB, there is illustrated an alternative overall operation of a music 
composition communications system, alternative to that illustrated in FIG. 2 A, showing 
additional automated modes including the network virtual performance mode A 5. FIG. 1 IB 
illustrates an alternative control logic structure for the network selection, where the decision 
logic proceeds in an analogous manner to FIG. 2 A. The operation starts at step 1100, and a 
mode is selected at step 1110. A composition is selected at 1120, and if the decision logic 1120 
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determines the composition has been selected, then processing proceeds at step 1130, which 
provides for selection of the mode. If no composition has been selected, processing proceeds 
back to the select composition step 1110. Once the composition has been selected, then a Mode 
is selected as illustrated at step 1130, and the decision logic of select Mode 1140 determines 
whether the selected mode is that of (1) a display mode, in which case processing proceeds to 
step 220 of FIG. 2 A, (2) a modified composition mode, in which case processing proceeds as 
from step 215 of FIG. 2A going to the yes branch, or (3) the select mode determines that the 
networked virtual performance mode has been selected, in which case processing proceeds to the 
networked mode automated Mode 5, step 1150, as illustrated in further detail in FIG. 12. 

Referring to FIG. 12, there is illustrated the networked virtual performance process flow, 
step 1150 of FIG. 1 IB, in greater detail. At step 1150, the automated mode of networked virtual 
performance has been selected. This automated mode 5 is processed by proceeding to verify the 
selection of the composition. At step 1220, the user's performance is captured (e.g., by a 
microphone or via MIDI) and converted to individual workstation performance data. This data is 
then broadcast by each of the plurality of individual music stands to a designated master 
controller. At step 1230, the controller receives and combines the plurality user's individual 
performance data to generate a combined performance data representative of the synchronized 
and combined plurality of individual performance data. This combined performance data is then 
broadcast to all of the individual music workstations. At step 1240, each of the plurality of 
individual music workstations provides an output (e.g., audio) responsive to the combined 
performance data, which is representative of the combination of the plurality of users audio 
performances of the music composition, synchronized to the display presentation of the music 
composition. Timing synchronization data is provided to synchronize the plurality of individual 
music workstations individual performance data to one another and to the selected musical 
composition's display presentation, to facilitate synchronization for purposes of combination and 
audio playback at the individual workstations, in approximately real time, so as to provide 
apparent real time feedback of the individual user's performance in combination with all other 
individual users, played back at each of the individual workstations. At step 1250, analysis is 
done as to whether the composition is done. If so, processing completes at step 1260 and ends 
this automated Mode 5. Otherwise, if processing is not done, that is the musical composition is 
not over, then processing proceeds back to step 1220 as illustrated. 
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Referring to FIG. 13, there is illustrated automated Mode 6 corresponding to 
synchronizing the actual display presentation to the user's performance, providing 
^synchronization of the display of the music to account for the actual performer's timing, and 
any errors or differences. Automated Mode 6 is analogous to A Mode 1 without override, but 
not to A Mode 3, which auto-advances independent of the user's performance. At step 1153, the 
automated modes 6 is entered. At step 1310, analogous to step 272 in FIG. 2D, the user's 
performance is tracked and stored for comparison to the musical composition display 
presentation relative to the timing of where the performance should be. At step 1320, the 
performance that has been tracked is compared to the score and deviation performance data for 
that performer is generated. At step 1330, the errors are utilized to resynchronize the display 
presentation of the musical composition to adjust to the actual user's real time performance 
timing relative to the display presentation and a synchronization of the commencement of the 
user's performance relative to the musical composition's presentation on the display. At step 
1340, an analysis is done as to whether processing is complete (e.g. is the musical composition 
done?). If so, A Mode 6 ends at processing step 1350, and either the automated modes of 
processing can end, or other automated modes can be checked for, and processing can begin for 
another selected automated mode. 

Referring to FIG. 14, there is illustrated the communications architecture and system in 
accordance with the present invention, wherein there are a plurality of individual music 
workstations 1410, each responsive to a user's performance of a musical input (1411) which is 
processed by the individual music workstation 1410 to provide a performance data output 1412 
coupled for communication to a control system 1450, which can be a master controller or master 
workstation, which is coupled for receiving and processing the individual workstation 
performance data for the plurality of individual workstations. A communications interface 1413, 
such as a modem, can provide for coupling and communication between the individual 
workstation 1410 and the master controller 1450. In one embodiment, the modems 1413 are 
coupled via a public switched telephone network to a corresponding modem 1455 which couples 
to the master workstation to communicate by directionally between the individual workstations 
and the master workstation. Alternatively, any sort of cable, hardwired, or wireless 
communications interface and provided, and would work equally well when in accordance with 
the present invention. The master controller 1450 is responsive to the individual performance 
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data outputs from the plurality of individual workstations to provide a combined and 
synchronized virtual performance data output 1451 which is coupled via the communications 
network to the plurality of individual workstations, which each then respectively provide an 
output (e.g. audio) 1415 for presentation to the user, representative of the combined performance 
data. An audio and/or visual presentation output 1452 can also be provided at the master 
workstation. 

Referring to FIG. 15, there is illustrated the data flow and communication architecture for 
one embodiment of the present invention. A music communication system is provided that has a 
plurality of physically separate remotely located locations, each having one or more individual 
music workstations, wherein each of the individual music workstations provides for 
communication output of individual performance data for a user's performance of a displayed 
musical composition at that individual workstation. As illustrated, a master controller 1505 is 
coupled to the communications infrastructure 1510 at a first location. A plurality of individual 
music workstations 1511, 1512, 1513, and 1514 are coupled to the communications 
infrastructure 1510 at location 1, designated performance group 1. Similarly, location 2, 
performance group 2, provides individual workstations 1521 and 1522 coupled to the 
communications infrastructure 1510, and location 3, performance 3, provides individual music 
workstation 1531, 1532, and 1533 coupled to the communications infrastructure 1510. The 
plurality of individual workstations 1511-21514, 1424-1521^1522 and 1531-1533, each provide 
individual performance data, providing multiple discrete time samples of performance data (e.g. 
time, segments of data), which can include synchronization information therein, which are 
communicated via the communication infrastructure 1510 to the controller or master workstation 
1505 The resultant output (e.g. audio) from the individual workstations is the resynchronized 
combined performance data of the plurality of individual performance data, resynchronized for 
each of the time segments of output. 

Referring to FIG. 16, the data word structure for one embodiment of the present invention 
is illustrated. The data structure for the individual user performance data is illustrated as 
comprising a header comprised of control information including a start of word, cyclic 
redundancy code (CRC) word, the performer's ID (a unique address for the individual 
workstation^ which not only can provide an unique ID but can also provide for indication of an 
instrument type), and other control information for other modes of operation in addition to the 
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virtual performance), payload data representing the actual performance data, and timing 
synchronization data. Any other data word structure providing for header information including 
control information, and payload information including performance data, and some timing 
synchronization scheme, whether or not part of the user performance data word will work 
equally well with the present invention. 

FIG. 17 illustrates a networked system embodiment in block diagram form, illustrating 
the network interface subsystem and music data processor subsystems for one embodiment of the 
present invention, compatible with the architecture as described in FIGS. 1-15. FIG. 17 
illustrates a single individual workstation 1710, although a plurality of individual workstations 
1710 are usually involved in the networked system (analogous to the plurality of individual 
workstations 1410 of FIG. 14). The individual workstation 1710 provides for communication 
with a master workstation 1790, (analogous to the communication architecture with the master 
workstation 1450 of FIG. 14). The musical performance by the user (1712) is coupled to an 
input interface 1715 which provides musical performance data to a musical data processor 1720, 
which can be a MIDI based processor, which provides the music processed performance data 
output to a network communications interface 1730 which outputs the individual musical 
performance data 1731 for coupling via the communications infrastructure to the master 
workstation 1790. 

The master workstation 1790 processes the individual musical performance data 1731, 
from the plurality of individual workstations 1710, and provides an output 1791 of combined 
virtual performance data, wherein both the individual musical performance data and the 
combined virtual performance data are comprised of a plurality of discrete time segmented data 
samples, wherein the combined virtual performance data is synchronized within each sample 
time to provide for synchronized combination of the individual musical performance for that 
time segment. 

A network interface subsystem 1735 is responsive to the combined virtual performance 
data to provide synchronized virtual performance data output to the music data processor 1725 
(e.g. MIDI) which provides a sound output of the combined virtual performance data 
corresponding to the synchronized combination of the individual musical performance data from 
a plurality of the individual workstations 1710. Additionally, the individual workstation 1710 
provides data analysis and presentation processing 1740 which provides an output responsive to, 
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inter alia, the combined virtual performance data to a presentation means such as an audiovisual 
display, or a video display or an audio output, 1745. Note that in a preferred embodiment, the 
multiple individual network workstations 1710 are synchronized (e.g. responsive to 
synchronization signal from the master workstation 1790), so that the plurality of the individual 
workstations simultaneously commence display presentation of the musical composition for 
synchronization of individual user performances. In one embodiment, 

the output of the individual performance data (such as via a MIDI music processor) is comprised 
of timing synchronization information that is used by the master workstation to synchronize all 
of the musical performances representative individual musical performance data to virtually 
perform together in a combined and time- synchronized performance. The combined virtual 
performance data representing the synchronized virtual performance is then sent back (e.g. 
broadcast) to the individual workstations, coupled via the network communication interface 1735 
back to the individual workstation's music data processor 1725, to provide audio and/or visual 
output corresponding thereto. Where the master workstation is for a conductor or teacher, the 
conductor mode or teacher mode can also be utilized in addition to the network virtual 
performance mod. In fact, all of the automated modes can be selectively combined, such as by 
providing layering of functions via software. 

Referring to FIG. 18, the timing diagrams for the individual performance data and 
synchronized virtual performance data is illustrated, relative to signals output from the individual 
networked workstations NWSi, NWS2, NWSn and a master workstation MWS. As illustrated, 
the master workstation MWS provides a synch signal to commence synchronization of all 
workstations (NWS) coupled to the system. In a preferred embodiment, the individual 
workstations (NWS) begin the display presentation of the musical composition (as does the 
master workstation) in synchronization with the synch signal from the master workstation. 
Responsive thereto, each of the individual workstations displays its musical composition, and the 
users of the workstations can perform the musical composition. Individual workstations (NWS) 
provide individual performance data responsive to the user's performance. As illustrated, 
workstation NWSl provides performance data Ar-a_and B-b_at respective times, while in the 
individual workstation NWS2 provides individual performance data C & D c and d at respective 
times, and independent workstation NWSn provides individual performance data signals E & F 
e and f at respective times. During the first time segment, individual performance dat a A, C & E 
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a, c and e are output, respectively, from individual workstations NWS ]_-K NWS? ^ and NWSnN. 
However, as shown in FIG. 18, the relative timing of output of the individual performance data 
(and receipt by the master workstation thereof) are skewed from one another in timing. The 
master workstation resynchronizes the individual performance data a, c and e A, C, and E within 
each time segment, and provides an output of the recombined reconstructed and synchronized a^ 
c and e A, C, and E to provide synchronized combined virtual performance data. It is to be noted 
that synchronized virtual performance data a, c and e A, C, E occurs within the first time segment 
Tl, which contains both the individual performance data and the corresponding responsive 
synchronized virtual performance data. The maximum time delay Td represents the acceptable 
worst case time that a user can tolerate between the user's performance output of corresponding 
individual performance data and the time until the synchronized virtual performance data 
corresponding to that individual performance data is received by the same individual 
workstation and provided as audio output. Similarly, during time slot 2, each of the individual 
workstations provides individual performance data b, d and f B, D and F , slightly time skewed 
from each other, and the master workstation provides combined synchronized virtual 
performance dat a b, d and f B, D, F , which is communicated back to the individual workstations 
to generate an audio output. 

The present invention finds application in many different environments and 
embodiments. For example, a live practice can be held from multiple locations, while one or 
more musicians are editing the score and communicating the changes real time, re-performing 
them, etc. An orchestra can be at one location, while a singer can be at another location. 
Creating and editing and synchronizing music to movies scores can also be facilitated with the 
present invention with beneficial gains. For example, in a movie, a lot of elements are changing 
dynamically that need to be resynchronized. For example, a scene can be lengthened or 
shortened, and the soundtrack must be adjusted. A score which was created to the movie must 
be rewritten to accommodate the edited movie. Typically, the composer creates the score at 
home or at a studio. That score is sent to the movie studio. However, the movie studio may edit 
the scene that music is used in, say a reduction of three seconds in the scene. The edited scene is 
sent back to the composer so that he can shorten his music. The process is slow, inefficient, and 
an impediment to the production process. By utilizing the present invention, this can be 
accommodated in real time, and from remote locations, and the virtual performance mode as well 
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as the other modes of the music communication of the present system can be utilized 
advantageously for this application. 

The present system provides an infrastructure for a music Internet. It can be applied to 
interactive editing of music. It can provide for special one on one mode for teaching and 
performance evaluation. It can facilitate adapting a music display to the performer's 
performance, or drive the display to a metronome and force the user's performance to be synched 
to the metronome timing. Music can be composed, edited, transposed, etc. easily, from any of 
the plurality of individual workstations or on a single standalone workstation. Multiple 
individual music workstations (IMW's) can communicate with one another, both with data and 
audio such as voice and/or music, and/or supplemental data such as video, audio, data files, 
control signals, applets, etc. Bands can practice at home without the disadvantage of either 
practicing alone or having to physically get together in a large grouping Singers can group with 
other instruments and other vocalists via respective ones of a plurality of IMW's, one, two, or 
even the whole plurality being remotely located physically from one another (e.g., LA, NY). 

There can be multiple instruments playing at a single location. There are many options 
regarding how IMW's are allocatable, such as. (1) on IMW (Individual Music Workstation) per 
location, (2) IMW per instrument type, (3) one IMW per performer (e.g. Instrument, vocal). 
There are also many options (1) for networked communications between the IMW's and/or 
between the IMW's and the Master Workstation. Modes can be selectively mixed and matched 
and combined, such as (1) all auto-modes enabled mode (2) virtual performance mode, full 
audiovisual, (3) audio link only mode, etc. There are also many options as to modes of signal 
processing which can be applied to the user input to generate the individual performance data, 
ranging from simple analog to digital (A/D), to complex musical sound analysis algorithms (e.g. 
to detect characteristics such as instrument type, pitch tone, attack, decay, amplitude, voice 
signatures, etc.) There are also tremendous numbers/parameters of performance analysis 
algorithms that can be selected and which utilize the selected musical composition data 
(corresponding to that displayed on the IMW which is receiving the performance input for 
analysis) (e.g. the sound characteristics generated by the sound analysis of the user's 
performance, to (3) in between "(1)" and "(2)". Additionally, many kinds of data can be 
communicated between (1) IMW's (2) IMW's and a Master Workstation, and (3) IMW's and 
external communication interfaces, etc. Examples of the kinds of data include performance data, 
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video data, processed/analyzed performance data, audio signals, data files, etc. 

Referring to FIG. 19, the memory work space structure for both the individual and the 
master workstations, in the virtual performance mode, is illustrated. Each of the individual 
workstations provides for music data buffering o the outgoing live performance data, individual 
performance data IPDXXX, and further provides for storage of the incoming combined virtual 
performance data CPDXXX. This buffering is done within read-write memory within the 
individual IMW's. Similarly, the data structure for the memory buffer of the master workstation 
is illustrated, showing separate buffer storage locations for each of the plurality of individual 
workstations WSi to WSn, providing storage of associated respective individual performance 
data IPD A-N. In one embodiment, multiple frame buffers are provided in the MWs, so that 
multiple time segments of EPD and CPD data can be stored in the buffer structure, and so that the 
buffer structure can support both input storage and output retrieval operations. Other data 
structures and buffering operations are also compatible with the present invention. The structure 
of the data word can be changed or adapted within its existing structure to facilitate the transfer 
of supplemental data is transferred from the IMWS or Master Workstation, or between the 
IMW's, or between the Imws and the Master Workstation. By utilizing the illustrated or other 
compatible data buffer structure, timing and synchronized (with the frame buffered variety) 
asynchronous communication are facilitated, not only as between communication workstations, 
but furthermore, as to the operations internally of the IMW. With frame buffering, the individual 
workstation and master workstation can dump the result output to the buffer, or receive input 
individual performance data asynchronously and buffered, relative to the other operations of the 
respective workstation, The IMW can facilitate operation in one mode or in multiple modes 
concurrently. 
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